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DATABASE AN NOTATION AND RRTRIEVAT. 

The present invention relates to the annotation of data 
files which are to be stored in a database for 
facilitating their subsequent retrieval. The present 
invention is also concerned with a system for generating 
the annotation data which is added to the data file and 
to a system for searching the annotation data in the 
database to retrieve a desired data file in response to 
a user's input query. 

Databases of information are well known and suffer from 
the problem of how to locate and retrieve the desired 
information from the database quickly and efficiently. 
Existing database search tools allow the user to search 
the_database using typed keywords. Whilst this is quick 
and efficient, this type of searching is not suitable for 
various kinds of databases, such as video or audio 
databases . 

According to one aspect, the present invention aims to 
provide a data structure which will allow the annotation 
of data files within a database which will allow a quick 
and efficient search to be carried out in response to a 
user's input query. 

According to one aspect, the present invention provides 
data defining a phoneme and word lattice for use as an 
annotation data for annotating data files to be stored 
within a database. Preferably, the data defines a 
plurality of nodes within the lattice and a plurality of 
links connecting the nodes within the lattice and further 
data associates a plurality of phonemes with a respective 
plurality of links and further data associates at least 
one word with at least one of said links. 
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According to another aspect , the present invention 
provides a method of searching a database comprising the 
annotation data discussed above, in response to an input 
query by a user- The method preferably comprises the 
steps of generating phoneme data and word data 
corresponding to the user's input query; searching the 
database using the word data corresponding to the user's 
query; selecting a portion of the data defining the 
phoneme and word lattice in the database for further 
searching in response to the results of the word search; 
searching said selected portion of the database using 
said phoneme data corresponding to the user ■ s input 
query; and outputting the search results. 



According to this aspect, the present invention also 
provides an apparatus for searching a database which 
employs the annotation data discussed above for 
annotating data files therein. The apparatus preferably 
comprises means for generating phoneme data and word data 
corresponding to a user ■ s input query; means for 
searching the database using the word data corresponding 
to the user's input query to identify similar words 
within the database; means for selecting a portion of the 
annotation data in the database for further searching in 
response to the results of the word search; means for 
searching the selected portion using the phoneme data 
corresponding to the user's input query; and means for 
outputting the search results. 



The phoneme and annotation data for a data file may be 
generated from the data file itself or from a typed or 
spoken annotation input by the user . 

Exemplary embodiments of the present invention will now 
be described with reference to the accompanying figures, 
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in which: 

Figure 1 is a schematic view of a computer which is 
programmed to operate an embodiment of the present 
5 invention; 

Figure 2 is a block diagram showing a phoneme and word 
annotator unit which is operable to generate phoneme and 
word annotation data for appendage to a data file; 

10 

Figure 3 is a block diagram illustrating one way in which 
the phoneme and word annotator can generate the 
annotation data from an input video data file; 

15 Figure 4a is a schematic diagram of a phoneme lattice for 
an example audio string from the input video data file; 

Figure 4b is a schematic diagram of a word and phoneme 
lattice embodying one aspect of the present invention, 
20 for an example audio string from the input video data 
file; 

Figure 5 is a schematic block diagram of a user's 
terminal which allows the user to retrieve information 
2 5 from the database by a voice query; 

Figure 6a is a flow diagram illustrating part of the flow 
control of the user terminal shown in Figure 5; 

30 Figure 6b is a flow diagram illustrating the remaining 
part of the flow control of the user terminal shown in 
Figure 5; 

Figure 7 is a flow diagram illustrating the way in which 
35 a search engine forming part of the user's terminal 
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carries out a phoneme search within the database; 

Figure 8 is a schematic diagram illustrating the form of 
a phoneme string and four M- GRAMS generated from the 
phoneme string ; 

Figure 9 is a plot showing two vectors and the angle 
between the two vectors; 

Figure 10 is a schematic diagram of a pair of word and 
phoneme lattices, for example audio strings from two 
speakers; 

Figure 11 is a schematic block diagram illustrating a 
15 user terminal which allows the annotation of a data file 
with^ annotation data generated from an audio signal input 
from a user; 



10 



20 



30 



Figure 12 is a schematic diagram of phoneme and word 
lattice annotation data which is generated for an example 
utterance input by the user for annotating a data file; 



Figure 13 is a schematic block diagram illustrating a 
user terminal which allows the annotation of a data file 
25 with annotation data generated from a typed input from a 



user ; 



Figure 14 is a schematic diagram of phoneme and word 
lattice annotation data which is generated for a typed 
input by the user for annotating a data file; 

Figure 15 is a block schematic diagram showing the form 
of a document annotation system; 

35 Figure 16 is a block schematic diagram of an alternative 
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document annotation system; 

Figure 17 is a block schematic diagram of another 
document annotation system; 

5 

Figure 18 is a schematic block diagram illustrating a 
user terminal which is operable to access a database 
located on a remote server via a data network in response 
to an input utterance by the user; 

10 

Figure 19 is a schematic block diagram of a user terminal 
which allows a user to access a database located in a 
remote server in response to an input utterance from the 
use^*; 

15 

Figure 20 is a schematic block diagram of a user terminal 
which allows a user to access a database by a typed input 
query; and 

20 Figure 21 is a schematic block diagram illustrating the 
way in which a phoneme and word lattice can be generated 
from script data contained within a video data fi^le. 

Embodiments of the present invention can be implemented 
25 using dedicated hardware circuits , but the embodiment to 
be described is implemented in computer software or code, 
which is run in conjunction with processing hardware such 
as a personal computer, work station, photocopier, 
facsimile machine, personal digital assistant (PDA) or 
30 the like. 

Figure 1 shows a personal computer (PC) 1 which is 
programmed to operate an embodiment of the present 
invention. A keyboard 3, a pointing device 5, a 
35 microphone 7 and a telephone line 9 are connected to the 
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PC 1 via an interface 11. The keyboard 3 and pointing 
device 5 enable the system to be controlled by a user. 
The microphone 7 converts acoustic speech signals from 
the user into equivalent electrical signals and supplies 
5 them to the PC 1 for processing. An internal modem and 
speech receiving circuit (not shown) is connected to the 
telephone line 9 so that the PC 1 can communicate with, 
for example, a remote computer or with a remote user. 

10. The programme instructions which make the PC 1 operate in 
accordance with the present invention may be supplied for 
use with an existing PC 1 on, for example, a storage 
device such as a magnetic disc 13, or by downloading the 
software from the Internet (not shown) via the internal 

15 modem and telephone line 9, 

DATA FILE ANNOTATION 

Figure 2 is a block diagram illustrating the way in which 
annotation data 21 for an input data file 2 3 is generated 

20 in this embodiment by a phoneme and word annotating unit 
25. As shown, the generated phoneme and word annotation 
data 21 is then combined with the data file 23 in the 
data combination unit 2 7 and the combined data file 
output thereby is input to the database 29. In this 

25 embodiment, the annotation data 21 comprises a combined 
phoneme (or phoneme like) and word lattice which allows 
the user to retrieve information from the database by a 
voice query. As those skilled in the art will 
appreciate, the data file 23 can be any kind of data 

30 file, such as, a video file, an audio file, a multimedia 
file etc. 

*> 

A system has been proposed to generate N-Best word lists 
for an audio stream as annotation data by passing the 
35 audio data from a video data file through an automatic 
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As shown in Figure 3, in this embodiment, the audio data 
31-2 is input to an automatic speech recognition unit 33, 
which is operable to generate a phoneme lattice 
corresponding to the stream of audio data 31-2. Such an 
automatic speech recognition unit 33 is commonly 
available in the art and will not be described in further 
detail. The reader is referred to, for example, the book 
entitled 'Fundamentals of Speech Recognition 1 by Lawrence 
Rabiner and Biing-Hwang Juang and, in particular, to 
pages 42 to 50 thereof, for further information on this 
type of speech recognition system. 

Figure 4a illustrates the form of the phoneme lattice 
data output by the speech recognition unit 33, for the 
input audio corresponding to the phrase * . . .tell me about 
Jason*.. . 1 . As shown, the automatic speech recognition 
unit 3 3 identifies a number of different possible phoneme 
strings which correspond to this input audio utterance. 
For example, the speech recognition system considers that 
the first phoneme in the audio string is either a /t/ or 
a /d/. As is well known in the art of speech 
recognition, these different possibilities can have their 
own weighting which is generated by the speech 
recognition unit 33 and is indicative of the confidence 
of the speech recognition unit's output. For example, 
the phoneme /t/ may be given a weighting of 0.9 and the 
phoneme /d/ may be given a weighting of 0.1, indicating 
that the speech recognition system is fairly confident 
that the corresponding portion of audio represents the 
phoneme /t/, but that it still may be the phoneme /d/. 
In this embodiment, however, this weighting of the 
phonemes is not performed. 

As shown in Figure 3, the phoneme lattice data 35 output 
by the automatic speech recognition unit 33 is input to 
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a word decoder 37 which is operable to identify possible 
words within the phoneme lattice data 35 . In this 
embodiment, the words identified by the word decoder 37 
are incorporated into the phoneme lattice data structure. 
5 For example, for the phoneme lattice shown in Figure 4a , 
the word decoder 37 identifies the words ■tell', 'dell 1 , 
' term ■ , ' me 1 , 1 a • , ' boat ■ , ' about ■ , • chase ' and ■ sun ' . 
As shown in Figure 4b, these identified words are added 
to the phoneme lattice data structure output by the 

10 speech recognition unit 33, to generate a phoneme and 
word lattice data structure which forms the annotation 
data 31-3. This annotation data 31-3 is then combined 
with the video data file 31 to generate an augmented 
video data file 31' which is then stored in the database 

15 29. As those skilled in the art will appreciate, in a 
similar way to the way in which the audio data 31-2 is 
time synchronised with the video data 31-1, the 
annotation data 31-3 is also time synchronised and 
associated with the corresponding video data 31-1 and 

20 audio data 31-2, so that a desired portion of the video 
and audio data can be retrieved by searching for and 
locating the corresponding portion of the annotation data 
31-3. 



2 5 In this embodiment, the annotation data 31-3 stored in 
the database 29 has the following general form: 
HEADER 

- time of start 

- flag if word if phoneme if mixed 

30 - time index associating the location of 

blocks of annotation data within memory to 
a given time point. 

- word set used (i.e. the dictionary) 

- phoneme set used 

35 - the language to which the vocabulary 



WO 00/54168 



PCT/GBOO/00718 



11 

pertains 

Block(i) i = 0,1,2, 

node j = 0, 1,2, 

- time offset of node from start of block 
5 - phoneme links (k) k = 0,1,2 

offset to node Nj = N k -N 3 (N k is node to 
which link K extends) or if N k is in 
block (i+1) offset to node Nj = N k +N b -Nj 
(where N b is the number of nodes .in 
10 block (i)) 

phoneme associated with link (k) 

- word links (1) 1 = 0,1,2, 

offset to node Nj = N A - Nj (Nj is node 
to which link 1 extends) or if N k is in 

15 block (i+1) offset to node Nj = N k +N b -Nj 

„ . (where N b is the number of nodes in 

block (i)) 

word associated with link (1) 

20 The time of start data in the header can identify the 
time and date of transmission of the data. For example, 
if the video file is a news broadcast, then the time of 
start may include the exact time of the broadcast and the 
date on which it was broadcast. 

25 

The flag identifying if the annotation data is word 
annotation data, phoneme annotation data or if it is 
mixed is provided since not all the data files within the 
database will include the combined phoneme and word 
30 lattice annotation data discussed above, and in this 
case, a different search strategy would be used to search 
this annotation data. 

In this embodiment, the annotation data is divided into 
35 blocks in order to allow the search to jump into the 
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middle of the annotation data for a given audio data 
stream. The header therefore includes a time index which 
associates the location of the blocks of annotation data 
within the memory to a given time offset between the time 
5 of start and the time corresponding to the beginning of 
the block. 

The header also includes data defining the word set used 
(i.e. the dictionary), the phoneme set used and the 
10 language to which the vocabulary pertains - The header 
may also include details of the automatic speech 
recognition system used to generate the annotation data 
and any appropriate settings thereof which were used 
during the generation of the annotation data. 

15 

Th^_blocks of annotation data then follow the header and 
identify, for each node in the block, the time offset of 
the node from the start of the block, the phoneme links 
which connect that node to other nodes by phonemes and 

20 word links which connect that node to other nodes by 
words. Each phoneme link and word link identifies the 
phoneme or word which is associated with the link. They 
also identify the offset to the current node. For 
example, if node N 50 is linked to node N 55 by a phoneme 

25 link, then the offset to node N 50 is 5. As those skilled 
in the art will appreciate, using an offset indication 
like this allows the division of the continuous 
annotation data into separate blocks. 

30 In an embodiment where an automatic speech recognition 
unit outputs weightings indicative of the confidence of 
the speech recognition units output, these weightings or 
confidence scores would also be included within the data 
structure. In particular, a confidence score would be 

35 provided for each node which is indicative of the 
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confidence of arriving at the node and each of the 
phoneme and word links would include a transition score 
depending upon the weighting given to the corresponding 
phoneme or word. These weightings would then be used to 
5 control the search and retrieval of the data files by 
discarding those matches which have a low confidence 
score. 

DATA FILE RETRIEVAL 

10 Figure 5 is a block diagram illustrating the form of a 
user terminal 59 which can be used to retrieve the 
annotated data files from the database 29. This user 
terminal 5 9 may be, for example, a personal computer, 
hand held device or the like. As shown, in this 

15 embodiment, the user terminal 59 comprises the database 
29 . jpf annotated data files, an automatic speech 
recognition unit 51, a search engine 53, a control unit 
55 and a display 57. In operation, the automatic speech 
recognition unit 51 is operable to process an input voice 

20 query from the user 39 received via the microphone 7 and 
the input line 61 and to generate therefrom corresponding 
phoneme and word data. This data may also take the form 
of a phoneme and word lattice, but this is not essential. 
This phoneme and word data is then input to the control 

25 unit 55 which is operable to initiate an appropriate 
search of the database 29 using the search engine 53. 
The results of the search, generated by the search engine 
53, are then transmitted back to the control unit 55 
which analyses the search results and generates and 

30 displays appropriate display data to the user via the 
display 57. 

Figures 6a and 6b are flow diagrams which illustrate the 
way in which the user terminal 59 operates in this 
35 embodiment. In step si, the user terminal 59 is in an 
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idle state and awaits an input query from the user 39. 
Upon receipt of an input query, the phoneme and word data 
for the input query is generated in step s3 by the 
automatic speech recognition unit 51. The control unit 
5 55 then instructs the search engine 53, in step s5, to 
perform a search in the database 29 using the word data 
generated for the input query. The word search employed 
in this embodiment is the same as is currently being used 
in the art for typed keyword searches, and will -not be 
10 described in more detail here. If in step s7, the 
control unit 55 identifies from the search results, that 
a match for the user's input query has been found, then 
it outputs the search results to the user via the display 
57. 

15 

In this embodiment, the user terminal 59 then allows the 
user to consider the search results and awaits the user's 
confirmation as to whether or not the results correspond 
to the information the user requires. If they are, then 

20 the processing proceeds from step sll to the end of the 
processing and the user terminal 59 returns to its idle 
state and awaits the next input query. If, however, the 
user indicates (by, for example, inputting an appropriate 
voice command) that the search results do not correspond 

25 to the desired information, then the processing proceeds 
from step sll to step s!3, where the search engine 53 
performs a phoneme search of the database 29. However, 
in this embodiment, the phoneme search performed in step 
sl3 is not of the whole database 29, since this could 

30 take several hours depending on the size of the database 
29 . 

Instead, the phoneme search performed in step sl3 uses 
the results of the word search performed in step s5 to 
35 identify one or more portions within the database which 
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may correspond to the user's input query. The way in 
which the phoneme search performed in step sl3 is 
performed in this embodiment, will be described in more 
detail later- After the phoneme search has been 
5 performed, the control unit 55 identifies, in step sl5, 
if a match has been found. If a match has been found, 
then the processing proceeds to step sl7 where the 
control unit 55 causes the search results to be displayed 
to the user on the display 57. Again, the system then 

10 awaits the user's confirmation as to whether or not the 
search results correspond to the desired information. If 
the results are correct, then the processing passes from 
step s!9 to the end and the user terminal 59 returns to 
its idle state and awaits the next input query. If 

15 however, the user indicates that the search results do 
not correspond to the desired information, then the 
processing proceeds from step sl9 to step s21, where the 
control unit 55 is operable to ask the user, via the 
display 57, whether or not a phoneme search should be 

20 performed of the whole database 29. If in response to 
this query, the user indicates that such a search should 
be performed, then the processing proceeds to stftep s2 3 
where the search engine performs a phoneme search of the 
entire database 29. 

25 

On completion of this search, the control unit 55 
identifies, in step s25, whether or not a match for the 
user's input query has been found. If a match is found, 
then the processing proceeds to step s2 7 where the 

30 control unit 55 causes the search results to be displayed 
to the user on the display 57. If the search results are 
correct, then the processing proceeds from step s29 to 
the end of the processing and the user terminal 59 
returns to its idle state and awaits the next input 

35 query. If, on the other hand, the user indicates that the 
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search results still do not correspond to the desired 
information, then the processing passes to step s31 where 
the control unit 55 queries the user, via the display 57, 
whether or not the user wishes to redefine or amend the 
5 search query. If the user does wish to redefine or amend 
the search query, then the processing returns to step s3 
where the user's subsequent input query is processed in 
a similar manner. If the search is not to be redefined 
or amended, then the search results and the .user's 
10 initial input query are discarded and the user terminal 
5 9 returns to its idle state and awaits the next input 
query. 

PHONEME SEARCH 

15 As mentioned above, in steps sl3 and s23, the search 
engine 53 compares the phoneme data of the input query 
with the phoneme data in the phoneme and word lattice 
annotation data stored in the database 2 9 . Various 
techniques can be used including standard pattern 

20 matching techniques such as dynamic programming, to carry 
out this comparison. In this embodiment, a technique 
which we refer to as M— GRAMS is used. This technique was 
proposed by Ng, K. and Zue, V.W. and is discussed in, for 
example, the paper entitled "Subword unit representations 

25 for spoken document retrieval" published in the 
proceedings of Eurospeech 1997, 

The problem with searching for individual phonemes is 
that there will be many occurrences of each phoneme 

30 within the database. Therefore , an individual phoneme on 
its own does not provide enough discriminability to be 
able to match the phoneme string of the input query with 
the phoneme strings within the database. Syllable sized 
units, however, are likely to provide more 

35 discriminability, although they are not easy to identify. 
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The M— GRAM technique presents a suitable compromise 
between these two possibilities and takes overlapping 
fixed size fragments, or M-GRAMS , of the phoneme string 
to provide a set of features. This is illustrated in 
5 Figure 8, which shows part of an input phoneme string 
having phonemes a, b, c, d, e, and f, which are split 
into four M-GRAMS (a, b, c), (b, c, d) , (c, d, e) and (d, 
e, f). In this illustration, each of the four M-GRAMS 
comprises a sequence of three phonemes which is unique 
10 and represents a unique feature (fi) which can be found 
within the input phoneme string. 

Therefore, referring to Figure 7, the first step s51 in 
performing the phoneme search in step sl3 shown in Figure 

15 6, is to identify all the different M-GRAMS which are in 
the input phoneme data and their frequency of occurrence* 
Then, in step s53, the search engine 53 determines the 
frequency of occurrence of the identified M-GRAMS in the 
selected portion of the database (identified from the 

20 word search performed in step s5 in Figure 6). To 
illustrate this, for a given portion of the database and 
for the example M-GRAMS illustrated in Figure 8^ this 
yields the following table of information: 



25 



M-GRAM 
( feature ( f t ) ) 


Input phoneme 
string frequency 
of occurrence 

(a) 


Phoneme string 
of selected 
portion of 
database 
(a) 


Mi 


1 


0 


M 2 


2 


2 


M 3 


3 


2 


M< 


1 


1 
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Next, in step s55, the search engine 53 calculates a 
similarity score representing a similarity between the 
phoneme string of the input query and the phoneme string 
of the selected portion from the database. In this 
5 embodiment, this similarity score is determined using a 
cosine measure using the frequencies of occurrence of the 
identified M- GRAMS in the input query and in the selected 
portion of the database as vectors- The philosophy 
behind this technique is that if the input phoneme -string 

10 is similar to the selected portion of the database 
phoneme string, then the frequency of occurrence of the 
M— GRAM features will be similar for the two phoneme 
strings. Therefore, if the frequencies of occurrence of 
the M— GRAMS are considered to be vectors (i.e. 

15 considering the second and third columns in the above 
tabl^ as vectors), then if there is a similarity between 
the input phoneme string and the selected portion of the 
database, then the angle between these vectors should be 
small. This is illustrated in Figure 9 for two- 

20 dimensional vectors a and q, with the angle between the 
vectors given as 0 . In the example shown in Figure 8, 
the vectors a and q will be four dimensional vectors and 
the similarity score can be calculated from: 



SCORE = COS 0 = =^ — (1) 

lal |al 

25 This score is then associated with the current selected 
portion of the database and stored until the end of the 
search. In some applications, the vectors used in the 
calculation of the cosine measure will be the logarithm 
of these frequencies of occurrences, rather than the 

30 frequencies of occurrences themselves. 



WO 00/54168 



PCT/GB00/00718 



19 

The processing then proceeds to step s57 where the search 
engine 53 identifies whether or not there are any more 
selected portions of phoneme strings from the database 
29. If there are, then the processing returns to step 
5 s53 where a similar procedure is followed to identify the 
score for this portion of the database. If there are no 
more selected portions , then the searching ends and the 
processing returns to step sl5 shown in Figure 6, where 
the control unit considers the scores generated by the 
10 search engine 53 and identifies whether or not there is 
a match by, for example, comparing the calculated scores 
with a predetermined threshold value. 

As those skilled in the art will appreciate, a similar 
15 matching operation will be performed in step s23 shown in 
Figure 6. However, since the entire database is being 
searched, this search is carried out by searching each of 
the blocks discussed above in turn. 

20 ALTERNATIVE EMBODIMENTS 

As those skilled in the art will appreciate, this type of 
phonetic and word annotation of data files in a database 
provides a convenient and powerful way to allow a user to 
search the database by voice- In the illustrated 

25 embodiment, a single audio data stream was annotated and 
stored in the database for subsequent retrieval by the 
user. As those skilled in the art will appreciate, when 
the input data file corresponds to a video data file, the 
audio data within the data file will usually include 

30 audio data for different speakers. Instead of generating 
a single stream of annotation data for the audio data, 
separate phoneme and word lattice annotation data can be 
generated for the audio data of each speaker. This may 
be achieved by identifying, from the pitch or from 

35 another distinguishing feature of the speech signals, the 
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audio data which corresponds to each of the speakers and 
then by annotating the different speaker's audio 
separately. This may also be achieved if the audio data 
was recorded in stereo or if an array of microphones were 
5 used in generating the audio data, since it is then 
possible to process the audio data to extract the data 
for each speaker - 

Figure 10 illustrates the form of the annotation data in 

10 such an embodiment, where a first speaker utters the 

words "... this so" and the second speaker replies "yes". 

As illustrated, the annotation data for the different 

speakers' audio data are time synchronised, relative to 

each other, so that the annotation data is still time 

15 synchronised to the video and audio data within the data 

file,. In such an embodiment, the header information in 

the data structure should preferably include a list of 

the different speakers within the annotation data and, 

for each speaker, data defining that speaker's language r 

2 0 accent , dialect and phonetic set, and each block should 

identify those speakers that are active in the block. 

i 

In the above embodiments, a speech recognition system was 
used to generate the annotation data for annotating a 

25 data file in the database. As those skilled in the art 
will appreciate, other techniques can be used to generate 
this annotation data. For example, a human operator can 
listen to the audio data and generate a phonetic and word 
transcription to thereby manually generate the annotation 

30 data. 

In the above embodiments, the annotation data was 
generated from audio stored in the data file itself. As 
those skilled in the art will appreciate, other 
35 techniques can be used to input the annotation data. 
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Figure 11 illustrates the form of a user terminal 59 
which allows a user to input voice annotation data via 
the microphone 7 for annotating a data file 91 which is 
to be stored in the database 29. In this embodiment, the 
5 data file 91 comprises a two dimensional image generated 
by, for example, a camera. The user terminal 59 allows 
the user 39 to annotate the 2D image with ah appropriate 
annotation which can be used subsequently for retrieving 
the 2D image from the database 29. In this embodiment, 

10 the input voice annotation signal is converted, by the 
automatic speech recognition unit 51, into phoneme and 
word lattice annotation data which is passed to the 
control unit 55. In response to the user's input, the 
control unit 55 retrieves the appropriate 2D file from 

15 the database 29 and appends the phoneme and word 
annotation data to the data file 91. The augmented data 
file is then returned to the database 29. During this 
annotating step, the control unit 55 is operable to 
display the 2D image on the display 57 so that the user 

2 0 can ensure that the annotation data is associated with 
the correct data file 91. 

The automatic speech recognition unit 51 generates the 
phoneme and word lattice annotation data by (i) 

25 generating a phoneme lattice for the input utterance; 
(ii) then identifying words within the phoneme lattice; 
and (iii) finally by combining the two. Figure 12 
illustrates the form of the phoneme and word lattice 
annotation data generated for the input utterance 

30 "picture of the Taj-Mahal", As shown, the automatic 
speech recognition unit identifies a number of different 
possible phoneme strings which correspond to this input 
utterance. As shown in Figure 12, the words which the 
automatic speech recognition unit 51 identifies within 

35 the phoneme lattice are incorporated into the phoneme 
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lattice data structure. As shown, for the example phrase, 
the automatic speech recognition unit 51 identifies the 
words "picture", "of", "off", "the", "other", "ta", 
"tar", "jam", "ah", "hal", "ha" and "al". The control 
5 unit 55 is then operable to add this annotation data to 
the 2D image data file 91 which is then stored in a 
database 29. 

As those skilled in the art will appreciate," this 
10 embodiment can be used to annotate any kind of image such 
as x-rays of patients, 3D videos of, for example, NMR 
scans, ultrasound scans etc. It can also be used to 
annotate one-dimensional data, such as audio data or 
seismic data. 

15 

In the above embodiment , a data file was annotated from 
a voiced annotation. As those skilled in the art will 
appreciate, other techniques can be used to input the 
annotation. For example, Figure 13 illustrates the form 

20 of a user terminal 59 which allows a user to input typed 
annotation data via the keyboard 3 for annotating a data 
file 91 which is to be stored in a database 29. In this 
embodiment, the typed input is converted, by the phonetic 
transcription unit 75, into the phoneme and word lattice 

25 annotation data (using an internal phonetic dictionary 
(not shown)) which is passed to the control unit 55. In 
response to the user's input, the control unit 55 
retrieves the appropriate 2D file from the database 29 
and appends the phoneme and word annotation data to the 

30 data file 91. The augmented data file is then returned 
to the database 29. During this annotating step, the 
control unit 55 is operable to display the 2D image on 
the display 57 so that the user can ensure that the 
annotation data is associated with the correct data file 

35 91 . 
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Figure 14 illustrates the form of the phoneme and word 
lattice annotation data generated for the input utterance 
"picture of the Taj-Mahal". As shown in Figure 2, the 
phoneme and word lattice is an acyclic directed graph 
5 with a single entry point and a single exit point. It 
represents different parses of the user's input. As 
shown, the phonetic transcription unit 75 identifies a 
number of different possible phoneme strings which 
correspond to the typed input. 

10 

Figure 15 is a block diagram illustrating a document 
annotation system. In particular, as shown in Figure 15, 
a text document 101 is converted into an image data file 
by a document scanner 103. The image data file is then 

15 passed to an optical character recognition (OCR) unit 105 
which converts the image data of the document 101 into 
electronic text. This electronic text is then supplied 
to a phonetic transcription unit 107 which is operable to 
generate phoneme and word annotation data 109 which is 

20 then appended to the image data output by the scanner 103 
to form a data file 111. As shown, the data file 111 is 

•a 

then stored in the database 29 for subsequent retrieval. 
In this embodiment, the annotation data 109 comprises the 
combined phoneme and word lattice described above which 
25 allows the user to subsequently retrieve the data file 
111 from the database 29 by a voice query. 

Figure 16 illustrates a modification to the document 
annotation system shown in Figure 15. The difference 

30 between the system shown in Figure 16 and the system 
shown in Figure 15 is that the output of the optical 
character recognition unit 105 is used to generate the 
data file 113, rather than the image data output by the 
scanner 103. The rest of the system shown in Figure 16 

35 is the same as that shown in Figure 15 and will not be 
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described further. 

Figure 17 shows a further modification to the document 
annotation system shown in Figure 15. In the embodiment 
shown in Figure, 17 , the input document is received by a 
facsimile unit 115 rather than a scanner 103. The image 
data output by the facsimile unit is then processed in 
the same manner as the image data output by the scanner 
103 shown in Figure 15, and will not be described again. 

In the above embodiment, a phonetic transcription unit 
107 was used for generating the annotation data for 
annotating the image or text data. As those skilled in 
the art will appreciate, other techniques can be used. 
For example, a human operator can manually generate this 
annotation data from the image of the document itself. 

In the above embodiment, the database 2 9 and the 
automatic speech recognition unit were both located 
within the user terminal 59. As those skilled in the art 
will appreciate, this is not essential. Figure 18 
illustrates an embodiment in which the database '2 9 and 
the search engine 53 are located in a remote server 60 
and in which the user terminal 59 accesses and controls 
data files in the database 29 via the network interface 
units 6 7 and 6 9 and a data network 68 (such as the 
Internet) - In operation, the user inputs a voice query 
via the microphone 7 which is converted into phoneme and 
word data by the automatic speech recognition unit 51. 
This data is then passed to the control unit which 
controls the transmission of this phoneme and word data 
over the data network 68 to the search engine 53 located 
within the remote server 60. The search engine 53 then 
carries out the search in accordance with the received 
phoneme and word data or controls the manipulation of the 
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data flies (for example to control the playing, 
forwarding or rewinding of a video file) in accordance 
with the received phoneme and word data. The data 
retrieved from the database 29 or other data relating to 
5 the search is then transmitted back, via the data network 
68, to the control unit 55 which controls the display of 
appropriate data on the display 57 for viewing by the 
user 39. In this way it is possible to retrieve and 
control data files in the remote server 60 without using 
10 significant computer resources in the server (since it is 
the user terminal 5 9 which converts the input speech into 
the phoneme and word data) . 



In addition to locating the database 29 and the search 

15 engine 53 in the remote server 60, it is also possible to 
locjate the automatic speech recognition unit 51 in the 
remote server 60. Such an embodiment is shown in Figure 
19. As shown in this embodiment, the input voice query 
from the user is passed via input line 61 to a speech 

20 encoding unit 73 which is operable to encode the speech 
for efficient transfer through the data network 68. The 
encoded data is then passed to the control unit 55 which 
transmits the data over the network 68 to the remote 
server 60, where it is processed by the automatic speech 

25 recognition unit 51. The phoneme and word data generated 
by the speech recognition unit 51 for the input query is 
then passed to the search engine 53 for use in searching 
and controlling data files in the database 29. 
Appropriate data retrieved by the search engine 53 is 

30 then passed, via the network interface 69 and the network 
68, back to the user terminal 59. This data received 
back from the remote server is passed via the network 
interface unit 67 to the control unit 55 which generates 
and displays appropriate data on the display 57 for 

35 viewing by the user. 
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In the above embodiments, the user inputs his query by 
voice. Figure 20 shows an alternative embodiment in 
which the user inputs the query via the keyboard 3. As 
shown , the text input via the keyboard 3 is passed to 
5 phonetic transcription unit 75 which is operable to 
generate a corresponding phoneme string from the input 
text. This phoneme string together with the words input 
via the keyboard 3 are then passed to the control unit 55 
which initiates a search of database using the search 

10 engine 53. The way in which this search is carried out 
is the same as in the first embodiment and will not, 
therefore, be described again. As with the other 
embodiments discussed above, the phonetic transcription 
unit 75, search engine 53 and/or the database 29 may all 

1 5 be located in a remote server . 

In the first embodiment, the audio data from the data 
file 31 was passed through an automatic speech 
recognition unit in order the generate the phoneme 

2 0 annotation data. In some situations, a transcript of the 
audio data will be present in the data file. Such an 
embodiment is illustrated in Figure 21. In this 
embodiment, the data file 81 represents a digital video 
file having video data 81-1, audio data 81-2 and script 

25 data 81-3 which defines the lines for the various actors 
in the video film. As shown, the script data 81-3 is 
passed through a text to phoneme converter 83, which 
generates phoneme lattice data 85 using a stored 
dictionary which translates words into possible sequences 

30 of phonemes. This phoneme lattice data 85 is then 
combined with the script data 81-3 to generate the above 
described phoneme and word lattice annotation data 81-4. 
This annotation data is then added to the data file 81 to 
generate an augmented data file 81' which is then added 

35 to the database 29. As those skilled in the art will 
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appreciate, this embodiment facilitates the generation of 
separate phoneme and word lattice annotation data for the 
different speakers within the video data file, since the 
script data usually contains indications of who is 
5 talking. The synchronisation of the phoneme and word 
lattice annotation data with the video and audio data can 
then be achieved by performing a forced time alignment of 
the script data with the audio data using an automatic 
speech recognition system ( not shown ) . 

10 

In the above embodiments, a phoneme (or phoneme-like) and 
word lattice was used to annotate a data file. As those 
skilled in the art of speech recognition and speech 
processing will realise, the word "phoneme" in the 
15 description and claims is not limited to its linguistic 
meaning but includes the various sub-word units that are 
identified and used in standard speech recognition 
systems . 
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CLAIMS ; 



1. Data defining a phoneme and word lattice, the data 
comprising: 

5 data for defining a plurality of nodes within the 

lattice and a plurality of links connecting the nodes 
within the lattice; 

data associating a plurality of phonemes with a 
respective plurality of links; and 
10 data associating at least one word with at least one 

of said links. 



2 . Data according to any preceding claim, wherein said 
data defining said phoneme and word lattice is arranged 
15 in blocks of nodes* 

3. Data according to claim 1, further comprising data 
defining time stamp information for each of said nodes. 

4. Data according to claim 3, arranged in blocks of 
equal time duration - 

j 

5. Data according to claim 2 or 4 , further comprising 
data defining each blocks location within said database. 

25 

6. Data according to claim 3 or any claim dependent 
thereon, wherein said data defining a phoneme and word 
lattice is associated with further data defining a time 
sequential signal , and wherein said time stamp 

3 0 information is time synchronised with said time 

sequential signal . 



20 



7. Data according to claim 6, wherein said further data 
defines an audio and/or video signal. 

35 
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8- Data according to claim 7 , wherein said further data 
defines at least speech data and wherein said data 
defining said phoneme and word lattice is derived from 
said further data. 

9. Data according to claim 8, wherein said speech data 
comprises audio data and wherein said data defining said 
phoneme and word lattice is derived by passing said audio 
signal through an automatic speech recognition system. 

10. Data according to claim 8 or 9, wherein said speech 
data defines the parol of a plurality of speakers , and 
wherein said data defines a separate phoneme and word 
lattice for the parol of each speaker. 

11. -Data according to any preceding claim, further 
comprising data defining a weighting for the phonemes 
and/or words associated with said links. 

12. Data according to any preceding claim, wherein at 
least one of said nodes is connected to a plurality of 
other nodes by a plurality of links. 

13. Data according to claim 12, wherein at least one of 
said plurality of links connecting said node to said 
plurality of other nodes is associated with a phoneme and 
wherein at least one of said links connecting said node 
to said plurality of other nodes is associated with a 
word . 

14. A method of searching a database comprising data 
according to any preceding claim, in response to an input 
query, the method comprising the steps of: 

generating phoneme data and/or word data 
corresponding to the input query; 
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searching the phoneme and word lattice using the 
phoneme and/or word data generated for the input query; 
and 

outputting search results in dependence upon the 
results of said searching step, 

15. A method according to claim 14, wherein said 
searching step comprises the steps of : 

(i) searching the phoneme and word lattice using the 
word data generated for the user ' s input query to 
identify similar words within the phoneme and word 
lattice ; 

(ii) selecting one or more portions of the phoneme 
and word lattice for further searching in response to the 
results of said word search; and 

— ( iii) searching said one or more selected portions 
of the phoneme and word lattice using the phoneme data 
generated for the user's input query. 

16. A method according to claim 15, wherein the results 
of the word search are output to the user before the 
phoneme search is performed on the selected portions of 
the database . 

17. A method according to claim 16, wherein said phoneme 
search is only performed in response to a further input 
by the user in response to the outputting of the results 
from the word search. 

18. A method according to any of claims 15 to 17, 
wherein said phoneme search is carried out by identifying 
a number of features within the phoneme sequence 
corresponding to the user's input query and identifying 
similar features within the data defining said phoneme 
lattice within the database- 
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19. A method according to claim 18, wherein each of said 
features represents a unique sequence of phonemes within 
the phoneme data of the user's input query. 

5 20. A method according to claim 19, wherein said phoneme 

search employs a cosine measure to indicate the 
similarity between the phoneme data corresponding to the 
user's input query and the phoneme data within the 
database. 

10 

21. A method according to any of claims 14 to 20 , 
wherein said search results are output to a display. 

22. A method according to any of claims 14 to 21, 
15 wherein said input query by the user is input by voice, 

and wherein said step of generating phoneme data and word 
data employs an automatic speech recognition system. 

23. A method according to any of claims 14 to 21, 
wherein said input query is a typed input and wherein 
said step of generating phoneme data and word data 
employs a text-to-phoneme converter. 

24. An apparatus for searching a database comprising 
25 data according to any of claims 1 to 13, in response to 

an input query, the apparatus comprising: 

means for generating phoneme data and/or word data 
corresponding to the input query; 

means for searching the phoneme and word lattice 
30 using the phoneme and/or word data generated for the 

input query; and 

means for outputting search results in dependence 
upon the output from said searching means. 



20 



35 



25. An apparatus according to claim 24, wherein said 
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searching means comprises: 

( i ) means for searching the phoneme and word lattice 
using the word data generated for the user's input query 
to identify similar words within the phoneme and word 
lattice; 

(ii) means for selecting one or more portions of the 
phoneme and word lattice for further searching in 
response to the results of said word search; and 

(iii) means for searching said one or more selected 
portions of the phoneme and word lattice using the 
phoneme data generated for the user's input query. 

26 • An apparatus according to claim 25 , wherein said 
output means is operable to output the results of the 
word search to the user before the phoneme search is 
performed on the selected portions of the database. 

27. An apparatus according to claim 26, wherein said 
phoneme search is only performed in response to a further 
input by the user in response to the outputting of the 
results from the word search. 

28. An apparatus according to any of claims 25 to 27, 
wherein said phoneme search is carried out by identifying 
a number of features within the phoneme sequence 
corresponding to the user's input query and identifying 
similar features within the data defining said phoneme 
lattice within the database. 

29. An apparatus according to claim 28, wherein each of 
said features represents a unique sequence of phonemes 
within the phoneme data of the user ' s input query . 

30. An apparatus according to claim 29, wherein said 
phoneme search employs a cosine measure to indicate the 
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similarity between the phoneme data corresponding to the 
user's input query and the phoneme data within the 
database . 

31. An apparatus according to any of claims 24 to 30, 
wherein said output means comprises a display. 

32. An apparatus according to any of claims 24 to 31, 
wherein said input query by the user is a voice -query, 
and wherein said means for generating phoneme data and 
word data comprises an automatic speech recognition 
system which is operable to generate said phoneme data 
and a word decoder which is operable to generate said 
word data. 



33- An apparatus according to any of claims 24 to 31, 

wherein said input query is a typed query and wherein 
said means for generating phoneme data and word data 
comprises a text-to-phoneme converter which is operable 
20 to generate said phoneme data. 

34. An apparatus for generating annotation data for use 
in annotating a data file comprising audio data, the 
apparatus comprising : 
25 an automatic speech recognition system for 

generating phoneme data for audio data in the data file; 

a word decoder for identifying possible words within 
the phoneme data generated by the automatic speech 
recognition system; and 

generating means for generating annotation data by 
combining the generated phoneme data and the decoded 
words . 



35 



35. An apparatus for generating annotation data for use 
in annotating a data file comprising text data, the 
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apparatus comprising : 

a text to phoneme converter for generating phoneme 
data for text data in the data file; and 

generating means for generating annotation data by 
combining the phoneme data and words in the text data. 

36. An apparatus for generating annotation data for use 
in annotating a data file, the apparatus comprising: 

input means for receiving an input voice signal; 

speech recognition means for converting the input 
voice signal into phoneme data and words; and 

generating means for generating annotation data by 
combining the phoneme data and the words . 

37. An apparatus for generating annotation data for use 
in annotating a data file, the apparatus comprising: 

input means for receiving a typed input from a user; 

converting means for converting words in the typed 
input into phoneme data ; and 

generating means for generating annotation data by 
combining the phoneme data and words in the typed input. 

38. An apparatus for generating annotation data for use 
in annotating a data file, the apparatus comprising: 

means for receiving image data representative of 

text ; 

character recognition means for converting said 
image data into text data; 

converting means for converting words in the text 
data into phoneme data; and 

generating means for generating annotation data by 
combining the phoneme data and words in the text data. 

39. An apparatus according to any of claims 34 to 38, 
wherein said annotation data defines a phoneme and word 
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lattice and wherein said generating means comprises: 

(i) means for generating data defining a plurality 
of nodes within the lattice and a plurality of links 
connecting the nodes within the lattice; 

(ii) means for generating data associating a 
plurality of phonemes of the phoneme data with a 
respective plurality of links; and 

(iii) means for generating data associating at least 
one of the words with at least one of said links.. 

40. An apparatus according to claim 3 9 , wherein said 
generating means is operable to generate said data 
defining said phoneme and word lattice in blocks of said 
nodes . 

41. An apparatus according to claim 3 9 or 40, wherein 
said generating means is operable to generate data 
defining time stamp information for each of said nodes. 

42. An apparatus according to claim 41, wherein said 
generating means is arranged to generate said phoneme and 
word lattice data in blocks of equal time duration. 

43. An apparatus according to claim 40, 41 or 42, 
wherein said generating means is operable to generate 
data which defines each block's location within a 
database - 

44. An apparatus according to claim 41 or any claim 
dependent thereon, wherein said data file includes a time 
sequential signal, and wherein said generating means is 
operable to generate time stamp data which is time 
synchronised with said time sequential signal. 

45. An apparatus according to claim 44, wherein said 
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•time sequential signal is an audio and/or video signal. 

46. An apparatus according to claim 34 or any claim 
dependent thereon, wherein said audio data includes audio 

5 data which defines the parol of a plurality of speakers, 

and wherein said generating means is operable to generate 
data which defines separate phoneme and word annotation 
data for the parol of each speaker. 

47. An apparatus according to claim 35 or any claim 
dependent thereon, wherein said text data defines the 
parol of a plurality of speakers, and wherein said 
generating means is operable to generate data defining 
separate phoneme and word annotation data for the parol 
of each speaker. 

48. An apparatus according to claim 34 or any claim 
dependent thereon, wherein said speech recognition system 
is operable to generate data defining a weighting for the 

20 phonemes in the phoneme data. 

49. An apparatus according to claim 34 or any' claim 
dependent thereon, wherein said word decoder is operable 
to generate data defining a weighting for the words 

25 identified within said phoneme data. 

50. An apparatus according to claim 39 or any claim 
dependent thereon, wherein said means for generating data 
defining a plurality of nodes and a plurality of links is 

30 operable to define at least one node which is connected 

to a plurality of other nodes by a plurality of links. 

51. An apparatus according to claim 50, wherein at least 
one of said plurality of links connecting said node to 

35 said plurality of other nodes is associated with a 
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phoneme and wherein at least one of said links connecting 
said node to said plurality of other nodes is associated 
with a word. 

52. An apparatus according to claim 36 or any claim 
dependent thereon, wherein said speech recognition means 
is operable to generate data defining a weighting for the 
phonemes in the phoneme data . 

53. An apparatus according to claim 52, wherein said . 
speech recognition means is operable to generate data 
defining a weighting for the words within the word data. 

54. An apparatus according to claim 36 or 37 or any 
claim dependent thereon, further comprising means for 
associating said annotation data with said data file. 

55. An apparatus according to claim 37 or any claim 
dependent thereon, wherein said converting means 
comprises an automatic phonetic transcription unit which 
generates said phoneme data from words within the typed 
input . 

56. An apparatus according to claim 38 or any claim 
dependent thereon, wherein said converting means 
comprises an automatic phonetic transcription unit which 
generates said phoneme data from words within the text 
data output by said character recognition means. 

57. An apparatus according to claim 38 or any claim 
dependent thereon, further comprising means for 
associating said annotation data with either said image 
data representative of said text or with said text data. 

58. An apparatus according to claim 38 or any claim 
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dependent thereon, wherein said receiving means comprises 
a document scanner or a facsimile machine, 

59- A method of generating annotation data for use in 
annotating a data file comprising audio data, the method 
comprising the steps of: 

using an automatic speech recognition system to 
generate phoneme data for audio data in the data file; 

using a word decoder to identify possible -words 
within the phoneme data generated by the automatic speech 
recognition system; and 

generating annotation data by combining the 
generated phoneme data and the decoded words . 

60. A method of generating annotation data for use in 
annotating a data file comprising text data, the method 
comprising the steps of: 

using a text to phoneme converter to generate 
phoneme data for text data in the data file; and 

generating annotation data by combining the phoneme 
data and words in the text data. 

61. A method of generating annotation data for use in 
annotating a data file , the method comprising the steps 
of: 

receiving an input voice signal; 

processing the input voice signal using a speech 
recognition system to generate phoneme data and word data 
for the input voice signal ; and 

generating annotation data by combining the phoneme 
data and the word data generated for the input voice 
signal . 

62 . A method of generating annotation data for use in 
annotating a data f ile , the method comprising the steps 
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of: 

receiving a typed input; 

converting words in the typed input into phoneme 
data; and 

generating annotation data by combining the phoneme 
data and words in the typed input. 

63. A method of generating annotation data for use in 
annotating a data file, the method comprising the ^steps 
of: 

receiving image data representative of text; 
converting said image data into text data using a 
character recognition unit; 

converting words in the text data into phoneme data; 

and 

^generating annotation data by combining the phoneme 
data and words within the text data. 

64. A method according to any of claims 59 to 63, 
wherein said annotation data defines a phoneme and word 
lattice and wherein said generating step comprises the 
steps of: 

(i) generating data defining a plurality of nodes 
within the lattice and a plurality of links connecting 
the nodes within the lattice; 

(ii) generating data associating a plurality of 
phonemes of the phoneme data with a respective plurality 
of links; and 

(iii) generating data associating at least one of 
the words with at least one of said links. 

65. A method according to claim 64, wherein said 
generating step generates said data defining said phoneme 
and word lattice in blocks of said nodes - 
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66. A method according to claim 64 or 65, wherein said 
generating step generates data defining time stamp 
information for each of said nodes. 

67. A method according to claim 66, wherein said 
generating step generates said phoneme and word lattice 
data in blocks of equal time duration. 

68. A method according to claim 65, 66 or 67, wherein 
said generating step generates data which defines each 
block's location within a database. 

69. A method according to claim 66 or any claim 
dependent thereon, wherein said data file includes a time 
sequential signal, and wherein said generating step 
generates time stamp data which is time synchronised with 
said time sequential signal. 

70. A method according to claim 69, wherein said time 
sequential signal is an audio and/or video signal. 

71. A method according to claim 5 9 or any claim 
dependent thereon, wherein said audio data includes audio 
data which defines the parol of a plurality of speakers, 
and wherein said generating step generates data which 
defines separate phoneme and word annotation data for the 
parol of each speaker. 

72. A method according to claim 60 or any claim 
dependent thereon, wherein said text data defines the 
parol of a plurality of speakers, and wherein said 
generating step generates data defining separate phoneme 
and word annotation data for the parol of each speaker. 

73. A method according to claim 59 or any claim 
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dependent thereon, wherein said speech recognition system 
generates data defining a weighting for the phonemes 
associated with said links - 

74. A method according to claim 59 or any claim 
dependent thereon, wherein said word decoder generates 
data defining a weighting for the words associated with 
said links. 

75. A method according to claim 64 or any claim 
dependent thereon, wherein said step of defining a 
plurality of nodes and a plurality of links defines at 
least one node which is connected to a plurality of other 
nodes by a plurality of links. 

76 . _A method according to claim 75, wherein at least one 
of said plurality of links connecting said node to said 
plurality of other nodes is associated with a phoneme and 
wherein at least one of said links connecting said node 
to said plurality of other nodes is associated with a 
word. 

•St 

77. A method according to claim 61 or any claim 
dependent thereon, wherein said speech recognition system 
generates data defining a weighting for the phonemes 
associated with said links. 

78. A method according to claim 61 or any claim 
dependent thereon, wherein said speech recognition system 
generates data defining a weighting for the words 
associated with said links. 

79. A method according to claim 61 or 62 or any claim 
dependent thereon, further comprising the step of 
associating said annotation data with said data file. 
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80. A method according to claim 62 or any claim 
dependent thereon, wherein said converting step uses an 
automatic phonetic transcription unit which generates 
said phoneme data for words within the typed input. 

5 

81. A method according to claim 63 or any claim 
dependent thereon, wherein said step of converting words 
into phonemes uses an automatic phonetic transcription 
unit which generates said phoneme data for words within 

10 the text data output by said character recognition unit. 

82. A method according to claim 63 or any claim 
dependent thereon, further comprising the step of 
associating said annotation data with either said 

15 received image data or with said text data. 

83. A method according to claim 63 or any claim 
dependent thereon, wherein said receiving step uses a 
document scanner or a facsimile machine. 

20 

84. A method of searching a data file including 
annotation data in response to an input query, the method 
comprising the steps of: 

generating phoneme data and word data corresponding 
25 to the input query; 

searching the data file based on the phoneme data 
and/or the word data and the annotation data; and 

outputting search results in dependence upon the 
result of said searching step. 

30 

85. A method according to claim 84, wherein said 
annotation data defines a phoneme and word lattice 
comprising : 

(i) data for defining a plurality of nodes within 
35 the lattice and a plurality of links connecting the nodes 
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within the lattice; 

(ii) data for associating a plurality of phonemes of 
the phoneme data with a respective plurality of links; 
and 

(iii) data for associating at least one word with at 
least one of said links. 

86. A method for storing a data file into a database, 
the method comprising the steps of: 

combining the data file with annotation data 
corresponding; to the data file, the annotation data 
including phoneme data; and 

storing the data file with the annotation data. 

87. An apparatus for searching a data file including 
annotation data, in response to an input query, the 
apparatus comprising : 

means for generating phoneme data and word data 
corresponding to the input query; 

means for searching a data file based on the phoneme 
data and/or the word data and the annotation data; and 

means for outputting a search result in dependence 
upon the result of said searching means. 

88. An apparatus according to claim 87, wherein said 
annotation data defines a phoneme and word lattice, and 
comprises : 

(i) data defining a plurality of nodes within the 
lattice and a plurality of links connecting the nodes 
within the lattice; 

(ii) data associating a plurality of phonemes of the 
phoneme data with a respective plurality of links; and 

(iii) data associating at least one word with at 
least one of said links. 
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89. An apparatus for storing a data file into a 
database , the apparatus comprising : 

means for inputting the data file and annotation 
data corresponding to the data file, the annotation data 
including phoneme data; and 

means for storing the data file with the annotation 

data* 

90. A medium for storing a data file, the data- file 
comprising : 

an audio data; and 

an annotation data corresponding to the audio data, 
said annotation data including phoneme data. 

91. A medium for storing a data file, the data file 
comprising: 

video data; 

audio data corresponding to the video data; and 
annotation data corresponding to the audio data, the 
annotation data including phoneme data. 

92. A medium for storing a data file, the data* file 
comprising: 

text data ; and 

annotation data corresponding to the text data, said 
annotation data including phoneme data. 

93. Data including audio data and further comprising 
annotation data corresponding to the audio data, which 
annotation data includes phoneme data. 

94 . Data including video data and further comprising 
audio data corresponding to the video data and annotation 
data corresponding to the audio data, which annotation 
data includes phoneme data. 
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95, Data including text data, the data further 
comprising annotation data corresponding to the text 
data, which annotation data includes phoneme data. 

96- A data carrier carrying data according to any of 
claims 1 to 13 or processor implementable instructions 
for controlling a processor to implement the method of 
any one of claims 14 to 23 or 59 to 83 or 84 to 86. 

97. Processor implementable instructions for controlling 
a processor to implement the method of any one of claims 
14 to 23 or 59 to 83 or 84 to 86. 



WO 00/54168 



1/21 



PCT/GB00/00718 




WO 00/54168 



2/21 



PCT/GB00/00718 



L 



23 



DATA 
FILE 



25 



PHONEME 
AND WORD 
ANNOTATOR 



PHONEME 
AND WORD 
ANNOTATION 
DATA 



lr 



21 



DATA 
COMBINATION 
UNIT 



27 



DATABASE 



29 



Fig. 2 



WO 00/54168 



PCT/GB00/00718 



3/21 



co 




WO 00/54168 



PCT/GB00/00718 



4/21 




WO 00/54168 PCT/GBOO/00718 

5/21 



59 




DISPLAY 



57 



51 



AUTOMATIC 
SPEECH 
RECOGNITION 
UNIT 



CONTROL 
UNIT 



SEARCH 
ENGINE 



DATABASE 



55 



.53 



29 



Fig. 5 



WO 00/54168 



PCT/GB00/00718 



6/21 



N 




GENERATE PHONEME AND 
WORD DATA FOR INPUT 
QUERY 



•s5 



PERFORM WORD SEARCH 

IN DATABASE USING 
WORDS IN INPUT QUERY 




s13 



USING RESULTS OF WORD 
SEARCH, PERFORM 
PHONEME SEARCH OF 
SELECTED PORTIONS OF 
DATABASE 



© 



Fig. 6a 



DISPLAY RESULTS 
TO USER 



DISPLAY 
RESULTS TO 
USER 



s9 




s17 




© 



PCT/GB00/00718 



7/21 




s21 





Y 




s23 




r / 


PERFORM PHONEME 


SEARCH OF ENTIRE 


DATABASE 




I ^s25 



0 



s27 



DISPLAY 
RESULTS TO 
USER 



s29 




Fig. 6b 



PCT/GBOO/00718 



8/21 



START 



IDENTIFY M-GRAMS AND 
THEIR FREQUENCY OF 
OCCURRENCE IN INPUT 
QUERY 



s51 



DETERMINE FREQUENCY 

OF OCCURRENCE OF 
IDENTIFIED M-GRAMS IN 
SELECTED PORTION OF 
DATABASE 



CALCULATE SIMILARITY 
SCORE BETWEEN INPUT 
QUERY AND SELECTED 
PORTION USING 

DETERMINED 
FREQUENCIES OF 
OCCURRENCES 




s53 



s55 



s57 



Fig. 7 



WO 00/54168 



9/21 



PHONEME 
STRING 




WO 00/54168 



PCT/GB00/00718 



10/21 




WO 00/54168 



PCT/GB00/00718 



11/21 



59 



39 




DISPLAY 



57 



n 



61 



i 



51 



AUTOMATIC 
SPEECH 
RECOGNITION 
UNIT 



CONTROL 
UNIT 



DATA FILE 



55 



91 



29 



DATABASE 



Fig. 11 



WO 00/54168 



PCT/GB00/00718 




WO 00/54168 



PCT/GB00/00718 



13/21 




57 



DISPLAY 



PHONETIC 
TRANSCRIPTION 
UNIT 



-KEYBOARD 



59 



CONTROL 
UNIT - 



55 




DATABASE 



29 



Fig. 13 



WO 00/54168 



PCT/GB00/00718 




PCT/GB00/00718 



15/21 



101 



.103 



SCANNER 



« ► 



OCR 



DATA FILE 
IMAGE DATA 
ANNOTATION DATA 



-105 



PHONETIC 
TRANSCRIPTION 
UNIT 



107 



-109 



ANNOTATION 
DATA 



-111 




Fig. 15 



WO 00/54168 



PCT/G BOO/007 18 



16/21 



-101 



.103 



SCANNER 



,105 



OCR 



DATA FILE 
TEXT DATA 
ANNOTATION DATA 



PHONETIC 
TRANSCRIPTION 
UNIT 



•113 



-107 



,109 



ANNOTATION 
DATA 



DATABASE 



29 



Fig. 16 



WO 00/54168 



PCT/GB00/00718 



17/21 



115 



,105 



FACSIMILE 
UNIT 



OCR 



DATA FILE 
IMAGE DATA 
ANNOTATION DATA 



PHONETIC 
TRANSCRIPTION 
UNIT 



-113 



,107 



,109 



ANNOTATION 
DATA 



DATABASE 



29 



Fig. 17 



WO 00/54168 



PCT/GB00/00718 



18/21 



60 



SERVER 



SEARCH 
ENGINE 



53 



■^29 



DATABASE 



61 



AUTOMATIC 
SPEECH 
RECOGNITION 
UNIT 



NETWORK 
INTERFACE 
UNIT 



69 




NETWORK 
INTERFACE 
UNIT 



CONTROL UNIT 



Fig. 18 



WO 00/54168 



PCT/GBOO/00718 



19/21 



60 



SERVER 



SEARCH 
ENGINE 



53 



DATABASE 



29 



AUTOMATIC 
SPEECH 
RECOGNITION 
UNIT 



NETWORK 
INTERFACE 
UNIT 



.51 



69 




UNIT 



Fig. 19 



WO 00/54168 



PCT/GB00/00718 



20/21 




Fig. 20 



WO 00/54168 



PCT/GBOO/00718 



21/21 



81 



VIDEO DATA FILE 
A 



VIDEO DATA 



AUDIO DATA 



SCRIPT DATA 



81-1 



81-2 



81-3 



83 



TEXT TO 
PHONEME 
CONVERTER 



PHONEME AND 

WORD 
ANNOTATION 
DATA 



8V 



VIDEO DATA FILE 
VIDEO DATA ^ 
AUDIO DATA 
SCRIPT DATA ~ 
ANNOTATION DATA 



.81-1 

■81-2 
'81-3 



81-4 



85 



PHONEME 
DATA 



81-4 




Fig. 21 



(12) INTERNATIONAL APPLICATION PUBLISHED UNDER THE PATENT COOPERATION TREATY (PCT) 




(19) World Intellectual Property Organization 
International Bureau 

(43) International Publication Date (10) International Publication Number 

14 September 2000 (14.09.2000) PCT WO 00/54168 A3 



(51) International Patent Classification 7 : G06F 17/30, 
G10L 15/00 



Centre Europe Limited, I Occam Court, Occam Road, 
Surrey Research Park, Guildford, Surrey GU2 SYI (GB). 



(21) International Application Number: PCT/GB00/O07I8 (74) Agents: BERESFORD, Keith, Dents, Lewis eta!.: Beres- 

ford & Co., 2-5 Warwick Court, High Hoiborn, London 



(22) International Filing Date: I March 2000 (01.03.2000) 



WC1R5DJ (GB). 



(25) Filing Language: 



(26) Publication Language: 



English 



(30) 



Priority Data: 

9905201.1 

9905186.4 

9905160.9 

9905199.7 

9905187.2 



5 March 1999 (05.03.1999) GB 

5 March 1999 (05.03. 1999) GB 

5 March 1999 (05.03. 1999) GB 

5 March 1999 (05.03.1999) GB 

5 March 1999 (05.03.1999) GB 



English (8! ) Designated States (national): AE, AL, AM. AT, AU, AZ, 
BA, BB, BG. BR. BY. CA, CH, CN, CR, CU. CZ, DE, DK, 
DM, EE, ES, FI. GB, GD. GE^GH, GM, HR. HU, ID, IL, 
IN, IS, JP, ICE, KG, KJ> ICR, KZ. LC, LK, LR. LS, LT. LU, 
LV, MA, MD, MG, MK, MN, MW, MX, NO, NZ. PL, PT, 
RO. RU, SD, SE, SG, SI, SK, SL, TJ, TM, TR, TT, TZ, UA, 
UG, US, UZ, VN, YU, ZA, ZW. 



(71) Applicant (for all designated States except US): CANON 
KABUSHIKI KAISHA [JP/JP); 30-2 3-chome Shimo- 
marukbTOhta-Ku, Tokyo (JP). 

(72) Inventors; and 

(75) Inventors/Applicants (for US only): 

CHARLESWORTH, Jason, Peter, Andrew [GB/GB]; 
Canon Research Centre Europe Limited, 1 Occam Court, 
Occam Road, Surrey Research Park, Guildford, Surrey 
GU2 5YJ (GB). RAJAN, Jebu, Jacob [IE/GB]; Canon 
Research Centre Europe Limited, 1 Occam Court, Occam 
Road, Surrey Research Park, Guildford, Surrey GU2 5YJ 
(GB). GARNER, Philip, Neil [GB/GB], Canon Research 



(84) Designated States (regional): ARIPO patent (GH, GM, 
KE, LS. MW, SD, SL, SZ, TZ, UG, ZW), Eurasian patent 
(AM, AZ, BY, KG, KZ, MD, RU. TJ, TM), European patent 
(AT, BE. CH. CY, DE. DK, ES, FI, FR, GB, GR. IE, IT, LU, 
MC, NL, PT, SE), OAPI patent (BF, BJ, CF, CG, CI, CM, 
GA, GN, GW, ML, MR. NE, SN, TD, TG). 

Published: 

— With international search report. 

(88) Date of publication of the international search report: 

14 December 2000 

For two-letter codes and other abbreviations, refer to the "Guid- 
ance Notes on Codes and Abbreviations" appearing at the begin- 
ning of each regular issue of the PCT Gazette. 



(54) Title: DATABASE ANNOTATION AND RETRIEVAL 



^ABQUT 



SUN 




\/d/ 



ME "V 



BOAT 



DEL 



/a/ M/ 
CHASE 



3 

rH ► 

^-f time 

IT) 

^ (57) Abstract: A data structure is provided for annotating data files within a database. The annotation data comprises a phoneme 
and word lattice which allows the quick and efficient searching of data files within the database in response to a user's input query. 
Q The structure of the annotation data is such that it allows the input query to be made by voice and can be used for annotating various 
^ kinds of data files, such as audio data files, video data fdes, multimedia data files etc. The annotation data may be generated from 
the data files themselves or may be input by the user either from a voiced input or from a typed input. 



INTERNATIONAL SEARCH REPORT 



Inton nat Application No 

PCT/GB 00/00718 



A. CLASSIFICATION OF SUBJECT MATTER 

IPC 7 G06F17/30 G10L15/00 



According to International Patent Gassrfication (IPC) or to both national classification and IPC 



B. FIELDS SEARCHED 



Minimum documentation searched (classification system followed by classification symbols) 

IPC 7 G06F G10L 



Documentation searched other than minimum documentation to the extent that such documents are included in the fields searched 



Electronic data base consulted during the international search (name of data base and, where practical, search terms used) 

EPO-Internal , WPI Data, PAJ, INSPEC, COMPENDEX 



C. DOCUMENTS CONSIDERED TO BE RELEVANT 



Category * ' Citation of document, with indication, where appropriate, of the relevant passages 



Relevant to claim No. 



GAGNOULET C ET AL : "MAIRIEVOX: A 
VOICE-ACTIVATED INFORMATION SYSTEM" 
SPEECH COMMUNICATION.NL, ELSEVIER SCIENCE 
PUBLISHERS. AMSTERDAM, 
vol. 10, no. 1, 

1 February 1991 (1991-02-01), pages 23-31, 

XP000202912 

iSSN: 0167-6393 

page 24, left-hand column, line 14 - line 
19 

page 24, right-hand column, paragraph 2.2 
- paragraph 2.3; figure IB 

-/— 



LH 



Further documents are listed in the continuation of box C. 



0 



Patent family members are iisted in annex. 



* Special categories of cited documents : 

*A* document defining the general state of the an wnich is not 
considered to be of particular reievance 

*E" earlier document but published on or after the international 
filing date 

*L* document which may throw doubts on priority claim(si or 
which is cited to establish trie publication date of another 
citation or other special reason (as specified) 

"O" document referring to an oral ciscfosure. use. exhibition or 
other means 

"P" document published pnor to trie international filing date but 
later than the priority date claimed 



T" later document published after the international filing date 
or priority date and not in conflict with the application but 
cited to understand the principle or theory underlying the 
invention 

"X" document of particular refevarce; the claimed invention 
cannot be considered novel or cannot be considered to 
involve an inventive step when the document is taken alone 

"Y* document of particular relevance: the claimed invention 

cannot be considered to involve an inventive step when the 
document is combined with one or more other such docu- 
ments, such combination being obvious to a person skilled 
tn the art. 

*&" document member of tne same patent family 



Date of the actual completion of the international searcn 



29 August 2000 



Date of mailing ot the international searcn repon 



12/09/2000 



Name and mailing address o( tne ISA 

European Patent Office. P. 3. 5818 Paieotiaan 2 
NL - 2280 HV Puswiik 

Tel. (+O1-70) 3*0-2040. Tx. 31 651 epo nl. 
Fax: (01-70) 



Authorized officer 



Fournier, C 



Form PCT1SA/210 (second sfioetl i July ■ 992i 



page 1 of 2 



INTERNATIONAL SEARCH REPORT 



Intor xi a I Application No 

PCT/GB 00/00718 



C.(Conttnuatton) DOCUMENTS CONSJOEREO TO BE RELEVANT 



Category • Citation ot document, with in die a b on. where appropriate, ot the relevant passages 



Relevant to daim No. 



WITBROCK M J ET AL: "Using words and 
phonetic strings for efficient information 
retrieval from imperfectly transcribed 
spoken documents" 
PROCEEDINGS OF THE 1997 2ND ACM 
INTERNATIONAL CONFERENCE ON DIGITAL 
LIBRARIES; PHILADELPHIA, PA, USA, 'Online! 
23 - 26 July 1997, pages 30-35, 
XP002146047 

Retrieved from the Internet: 

<URL : http : //www . cs . emu . edu/af s/cs/u ser/al e 

x/docs/i dvl/dl 97 . html > 

'retrieved on 2000-08-29! 

abstract 

page 3, line 32 -page 4, line 6 



S. CASSIDY & J. HARRINGTON: "EMU: an 
Enhanced Hierarchical Speech Data 
Management System" 

PR0C. OF 6TH SPEECH SCIENCE AND TECHNOLOGY 
CONFERENCE, ADELAIDE, AUSTRALIA, 

1996, pages 381-386, XP002145977 
page 382, line 1 -page 383, line 6; figure 
2 



EP 0 649 144 A (IBM) 

19 April 1995 (1995-04-19) 



column 1, line 24 - line 30 

column 13. line 37 -column 14, line 8 

column 6, line 3 -column 12, line 25 



EP 0 597 798 A (IBM) 

18 May 1994 (1994-05-18) 



column 2, 1 i ne 35 - 1 ine 41 
column 4, line 9 - line 58 



14,24, 
34,36, 
59,61, 
84,86, 
87,89-97 

1-8,14, 
24, 

34-38, 
59-62 



84,86, 
87,89, 
90 , 93 , 
96,97 

1,3,5-9, 

14,24, 

34-38, 

59-62, 

84,86, 

87,89-97 



14, 

21-24, 
31-38, 
59-62 , 
84,87, 
89-97 



Perm PCT/1SA.2 



lO icooonuaaan oi secona sneeil (JUy ^992) 



page 2 of 



2 



INTERNATIONAL SEARCH REPORT 

information on patent family members 



Inter mat Application No 

PCT/GB 00/00718 



Patent document 
cited in search report 


Publication 
date 


Patent family 
member(s) 


Publication 
date 


EP 0649144 A 


19-04-1995 


DE 


69422466 


0 


10-02-2000 






DE 


69422466 


T 


06-07-2000 






JP 


2986345 


B 


06-12-1999 






JP 


7199379 


A 


04-08-1995 






US 


5649060 


A 


15-07-1997 



EP 0597798 A 18-05-1994 JP 2741833 B 22-04-1998 

JP 6274533 A 30-09-1994 



Form PCT1SA/210 (patent tamdy annexi (July 19921 



P.O. m^rl^z—x 

p. J&s: (^J 



XP-0021 92893 

" 175 



CONNECTIVITY 

Thus condition (T) is satisfied ; condition (2') is obviously satisfied 
2. Articulation vertices and blocks 



Q.E.D. 



A vertex whose removal from the graph increases x tjie number of connected 
components is called an articulation vertex. An edge whose removal from the 
graph increases the number of connected components of the graph is called an 
isthmus. Using these definitions, we can redefine 2-connectivity : A graph is 2- 
connected if, and only if it is of order n 3> 3, connected, and has no articulation 
vertices. 

A set A of vertices in graph G that generates a subgraph G A that is con- 
nected and without articulation vertices and is maximal with respect to this 




Fig. 9.2. Cactus 

property is called a block. Thus, subgraph G A is either 2-connected (if 
\ A \ > 2), or an isthmus of G (if | A | = 2), or an isolated vertex of G 
(if M| = 1). It is left to the reader to verify that the graph in Fig. 9 2 has 9 
articulation vertices, 6 isthmi, and 13 blocks. 

A chord of an elementary cycle is defined to be an edge that joins two non- 
consecutive vertices of the cycle. A cycle of length 3 has no chords A cycle 
of length 4 can have 0, 1 or 2 chords. 

A cactus is defined to be a connected graph in which every block is either an 
isthmus or an elementary cycle without chords (see Fig. 9.2). 

The principal characteristics of 2-connected graphs are described in the 
iollowmg theorem. 
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